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(54) A spread spectrum watermark for embedded signalling 



(57) A watermark is embedded into audio/video/- 
image/multimedia data using spread spectrum method- 
ology. The watermark is extracted from watermarked 
data without the use of an original or unwatermarked 



version of the data by using spatial or temporal local 
averaging of the frequency coefficients of the water- 
marked data. 



4 o 



WATERMARKED 
IMAGE 



OCT 



WATERMARKED 

XHACZ 

SPECTRUM 



4 4 



ORIGINAL 




IMAGE 







4 6 



OR1CIKAL 
SFBCTRUH 



4 2 



INSERTED 
WATERMARK 



EXTRACTED 
WATERMARX 



COMPARATOR 



5 0 



5 7 



STATISTICAL 
CONFIDENCE 
LEVEL 



CM 
< 
CM 

CO 

CO 
CM 
CO 

O 

CL 
LU 



^«n;rc:.v ■"?:<:••• »U»*: business St»v«c«* 



1 



EP 0 828 372 A2 



2 



Description 

Field of the Invention 

The present invention relates to digital watermark- 
ing of data including audio, video, image and multimedia 
data. Specifically, the invention relates to the extraction 
of a watermark of embedded data from watermarked 
data without using an original or unwatermarked version 
of the data. 

Background of the Invention 

The proliferation of digitized media such as video, 
image, video and multimedia is creating a need for a 
security system which facilitates the identification of the 
source of the material- 
Context providers, i.e. owners of works in digital 
data form, have a need to embed signals into 
audio/video/image/multimedia data which can subse- 
quently be recorded or detected by software and/or 
hardware devices for purposes of authenticating copy- 
right ownership, control and management. 

For example, a coded signal might be inserted in 
data to indicate that the data should not be copied. The 
embedded signal should preserve the image fidelity, be 
robust to common signal transformations and resistant 
to tampering. In addition, consideration must be given to 
the data rate that can be provided by the system, 
though current requirements are relatively low - a few 
bits per frame. 

In U.S. Patent Application 08/534,894, filed Sep- 
tember 28. 1995. entitled "Secure Spread Spectrum 
Watermarking for Multimedia Data" and assigned to the 
same assignee as the present invention, there was pro- 
posed a spread spectrum watermarking method which 
embedded a watermark signal into perceptually signifi- 
cant regions of an image for the purposes of identifying 
the content owner and/or possessor. A strength of this 
approach is that the watermark is very difficult to 
remove. In fact, this method only allows the watermark 
to be read if the original image or data is available for 
comparison. This is because the original spectrum of 
the watermark is shaped to that of the image through a 
non-linear multiplicative procedure and this spectral 
shaping must be removed prior to detection by matched 
filtering and the watermark is inserted into the N largest 
spectral coefficients, the ranking of which is not pre- 
served after watermarking. Thus, this method does not 
allow software and hardware devices to directly read 
embedded signals. 

In an article by Cox et al.. entitled " Secured Spec- 
trum Watermarking for Multimedia" spread spectrum 
watermarking is described which embed a pseudo-ran- 
dom noise sequence into the digital data for watermark- 
ing purposes. 

The prior art watermark extraction methodology 
requires the original image spectrum be subtracted 



from the watermark image spectrum. This restricts the 
use of the method when there is no original image or 
original image spectrum available. One application 
where this presents a significant difficulty is for third 
5 party device providers desiring to read embedded infor- 
mation for operation or denying operation of such a 
device. 

The present invention extends the earlier work of 
Cox et al to allow the reading or extraction of embedded 

ic signals by devices which do not contain original data, 
e.g. original images. 

In U.S Patent No. 5,319.735 by R.D. Preuss et al 
entitled "Embedded Signalling" digital information is 
encoded to produce a sequence of code symbols. The 

75 sequence of code symbols is embedded in an audio sig- 
nal by generating a corresponding code signal repre- 
senting the sequence of code symbols. The frequency 
components of the code signal being essentially con- 
fined to a preselected signalling band lying within the 

20 bandwidth of the audio signal and successive segments 
of the code signal corresponds to successive code sym- 
bols in the sequence. The audio signal is continuously 
frequency analyzed over a frequency band encompass- 
ing the signalling band and the code signal is dynami- 

2t cally filtered as a function of the analysis to provide a 
modified code signal with frequency component levels 
which are. at each time instant, essentially a prese- 
lected proportion of the levels of the audio signal fre- 
quency components in corresponding frequency 

so ranges. The modified code signal and the audio signal 
are combined to provide a composite audio signal in 
which the digital information is embedded. This compo- 
nent audio signal is then recorded on a recording 
medium or is otherwise subjected to a transmission 

35 channel. 

Summary of the Invention 

The present invention overcomes the limitations of 
40 the prior systems by using spread spectrum technology 
to embed watermark data or information into predeter- 
mined locations in an image. 

More specifically, the invention provides a system 
for extracting a watermark from watermarked data with- 
45 out using an original or unwatermarked version of the 
data. 

The preferred method of watermarking extraction is 
to use a spatial or temporal local average of the fre- 
quency coefficients of the watermarked data to deter- 

50 mine the watermark. The frequency coefficients of a 
two<limensional neighborhood in two-dimensional 
watermarked data (e.g. an image), for example, are 
analyzed to reproduce the entire watermark. This is 
possible since the watermark is embedded into the data 

55 using spread spectrum technology which places the 
watermark throughout the data. 

The invention is applicable to the watermarking of 
audio/video/image/multimedia data. 
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The invention will be best understood when the fol- 
lowing description is read in conjunction with the 
accompanying drawing. 

Brief Description of the Drawings s 

Figure 1 is a schematic block diagram of a method 
of inserting a watermark into an image; 

Figure 2 is a graphical representation of the image to 
spectrum and shaped watermark spectrum; 

Figure 3 is a schematic block diagram of a com- 
biner; 

75 

Figure 4 is a schematic block diagram of a method 
of extracting a watermark from a watermarked 
image; 

Figure 5 is a schematic block diagram of a separa- 20 
tor; 

Figure 6 is a schematic block diagram of a spread 
spectrum system for use in watermark insertion; 

25 

Figure 7 is a schematic block diagram of a spread 
spectrum receiver; 

Figure 8 is an original image to be watermarked; 

30 

Figure 9 is the image in Figure 8 after being water- 
marked; and 

Figure 10 shows a 4x4 array indicating the 
sequence of coefficients used to form a one-dimen- 35 
sional vector. 

Detailed Description 

Referring now to the figures and to Figure 1 in par- 40 
ticular, there is shown a schematic block diagram of a 
method for inserting a watermark into a digital data, for 
instance an image. In the following description refer- 
ence may be made to image data or images. While the 
invention has applicability to image data and images, it 45 
will be understood that the teachings herein and the 
invention itself are equally applicable to audio, video, 
image and multimedia data and the term image and 
image data will be understood to include these terms 
where applicable. As used here in watermark will be so 
understood to include embedded data, symbols, 
images, instructions or any other identifying mfoi mation 

The image 10 is first transformed into a spatial fre- 
quency representation 12. for instance by a disci ete 
cosine transform (DCT), other transforms such as a fast -f 
Fourier transform could also be used. The spectrum is 
then analyzed to determine the perceptually most sig- 
nificant components 14 and the watermark to be imbed - 
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ded 16 is then combined 18 with the perceptionally most 
significant components. The watermark is a pseudo 
random number sequence (PNS) preferably chosen 
from a Gaussian distribution. After being combined, the 
modified image is then inverse transformed back into 
the spatial domain to create the watermarked image 20. 

There are different ways to combine the watermark 
with the image spectrum. In the preferred embodiment, 
the watermark components, W h are added to the fre- 
quency coefficients, /,. in a non-linear manner as 

/,'=/, W,W, 0) 

where a is a constant typically in the range of 0.1 to 
0.01 . In principle, a might also vary as a function of fre- 
quency and perceptual modeling. Equation (i) can be 
considered a form of spectral shaping. That is . the orig- 
inal Gaussian white spectrum of the watermark is 
shaped to match that of the image by the second term 
in Equation (1) prior to addition ol the two spectra. The 
constant a serves as a gain control to adjust the relative 
strength of the two spectra. This is graphically shown in 
Figure 2. 

The two stages of the combiner are shown in Figure 
3. The watermark to be embedded into the data is pro- 
vided as a first input to a spectral shaper 30. The spec- 
trum of the image to be watermarked is provided as a 
second input to the shaper 30. The output of shaper 30 
is provided as a final input to summer 32. The spectrum 
of the image is provided as the second input to summer 
32. The output of summer 32 is a watermarked spec- 
trum. 

To extract the watermark, the inverse process must 
be applied as shown in Figure 4. The separator stage 
inverts the combiner stage. In order to extract the water- 
mark components, W h from a possibly distorted water- 
mark image, first subtract the original image before 
dividing by the image spectral coefficients. The latter 
process serves to normalize or equalize the watermark 
spectrum back to its original shape. That is 

w.MfrftVvf, (2) 

Specifically, the watermarked image 40 is trans- 
formed by a discrete cosine transform or other transfor- 
mation such at FFT, into a watermarked image 
spectrum 42. The stored original image 44 is trans- 
formed into an original image spectrum 46. 

The watermarked image spectrum 42 and the orig- 
inal image spectrum 46 are provided as inputs to sepa- 
rator 48. The separator, as shown in Figure 5. subtracts 
the original image spectrum from the watermarked 
image spectrum 54 to obtain a difference image spec- 
trum prior to normalizing the resultant image. The spec- 
tral normalization 56 divides the difference image 
spectrum by the image spectral coefficients uf, to yield 
an extracted watermark. 

The extracted watermark is statistically compared 
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with the known inserted watermark to calculate a statis- 
tical confidence level. The statistical confidence level 
provides a measure of whether the external watermark 
is the actual inserted watermark. 

In the above described method of extracting a 
watermark, it is necessary to have an original unwater- 
marked image. This is both an advantage and limitation 
of the method. It is advantageous because it is difficult 
for a non-possessor of the image to remove the water- 
mark. A limitation of the method is that it prevents a third 
party's software or hardware devices from extracting or 
reading the embedded signal information. The use of a 
Gaussian noise distribution is important for extraction 
using an original image. 

The above described prior art system is a special 
case of a more general spread spectrum communica- 
tion system in which the watermark information is con- 
sidered as the signal and the image is considered as the 
noise. Figure 6 is a schematic block diagram of a spe- 
cial spectrum communication system for use in water- 
mark insertion. 

In Figure 6 a watermark signal is provided as an 
input to an error correction encoder 60. The output of 
encoder 60 is provided to a spread spectrum modulator 
62. The output of modulator 62 is provided to a spectral 
transformation 64. The output of spectral transformation 
64 is provided as one input to a spectral shaper 66. A 
signal to be watermarked is provided to a spectral trans- 
former 68. The output of the transformer 68 is provided 
as a second input to spectral shaper 66 and to a delay 
70. The output of the spectral shaper 66 is added to the 
output of delay 70 at a summer 72. The second output 
is subject to an inverse transform 74. The result of the 
inverse transform is a watermarked signal. 

In the prior systems, the object was to embed a sin- 
gle PN (pseudo random number) sequence into an 
image. The information associated with the PN 
sequence was assumed to be stored in a database 
together with the original image and the spectral loca- 
tion of the embedded watermark. The locations of the 
watermarked components had to recorded because the 
implementation approximated the N perceptionally most 
significant regions of the watermark by the N largest 
coefficients. However, this ranking was not invariant to 
the watermarking process. The N largest coefficients 
may be different after inserting the watermark than 
before intersecting the watermark. 

In order to avoid this problem, the current method 
places a watermark in predetermined locations of the 
spectrum, typically the first N coefficients. However, any 
predetermined locations could be used even though 
such locations should belong to the perceptually signal 
cant regions of the spectrum if the watermark is to sur- 
vive common signals transformations such as 
compression, scaling, etc. 

More generally, the information to be embedded is 
a sequence of m symbols drawn from an alphabet A 
(e.g. the binary digits or the ASCII symbols). This data 



is then supplemented with additional symbols for error 
detection and correction. Each symbol is then spread 
spectrum modulated, a process that maps each symbol 
into a unique PN sequence known as a chip. The 

5 number of bits per chip is preset - the longer the chip 
length, the higher the detected signal-to-noise ratio will 
be. but this is at the expense of signaling bandwidth. 

The spectrum of the PN sequence is white, i.e. flat, 
and is therefore shaped to match that of the "noise", i.e. 

to the image/video/audio/or multimedia data into which the 
watermark is to be embedded. It is this spectral shaping 
that must be modified from the prior methods so that the 
extraction process no longer requires the original 
image. To do this, Equation (1) is modified so that each 

75 coefficient of the watermarked spectrum is scaled by 
the local average of the image spectral coefficient rather 
than the coefficient itself, i.e. 

//=/ J+ aav0(f,)W, (3) 

PC 

This average may be obtained in several ways. It 
may be a local average over a two dimensional region. 
Alternatively, the two dimensional spectrum may be 
sampled to form a one dimensional vector and a one 

25 dimensional local average may be performed. The latter 
method was used in experimental results below. The 
average may be a simple box or weighted average over 
the neighborhood. 

For video data, temporal averaging of the spectral 

30 coefficients over several frames can also be applied. 
However, since several frames are needed for averag- 
ing at the spectral normalization stage of the extractor, 
the protection of individual video frames taken in isola- 
tion may not be possible. For this reason, the present 

35 invention treats video as a very large collection of still 
images. In this way. even individual video frames are 
copy protected. 

Receiving or extracting a spread spectrum signal is 
shown in Figure 7. The watermarked image, video, 

<o audio or multimedia data is first spectrally normalized 
76 to undo any previously performed spectral shaping. 
The normalized signal is then analyzed by a bank of 
correlators 78A...78Z. each corrector detecting the 
presence, if any, of a particular PN sequence (one for 

45 each symbol in the alphabet). The decision circuit 80 
typically selects the correlator with the maximum output 
as the most likely current symbol. More sophisticated 
decision procedures are possible. The sequence of 
most likely current symbols is then provided as an input 
to an error correction stage 82 which corrects for faise 
decisions made by the decision circuit. The output of the 
error correction stage is an extracted watermark signal. 
In order to perform the spectral normalization 76, the 
previously performed spectral shaping procedure is 
•nveited. In the present case, the original unwater- 
marked signal is no longer available. Thus, an average 
of the frequency coefficients, avg(fj % as approximated 
by the average of the watermarked signal, i.e. avg(f,') 
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This is approximately true since the second term of 
equation (3) is small relative to the first i.e. 

aavB(/ i )W j «/ i (5) 

The normalization stage then divides each coeffi- 
cient (/,*) in the received signal by the local average avg 
(f$l « n the neighborhood. 

That is. 

f£ J^aav9(f,)W, 
avo(/,f avg(f{) 

f 

The first term, on the tight hand side (RHS) of 
Equation (6) . 

L 



Equation <6) which is not present when using the prior 
art method. 

In order to determine the closeness of the approxi- 
mation of Equation (4), the equalization process was 

f repeated using the original shaping coefficients, i.e. 
avg(fj t instead of avg (/,"). The correlator response 
increase from 0.125 to 0.15. suggesting that a loss of 
approximately 20% was incurred due to this approxima- 
tion. Of course, this loss is strongly dependent of the 

70 local smoothness in the spectra of an image and may 
vary significantly from image to image. 

In summary, the present invention provides a modi- 
fication to existing digital watermarking methods in 
which the original data was required for watermark 

75 extraction thereby enabling watermarking extraction in 
the absence of an unwatermarked or original data. The 
present invention uses local spatial and/or temporal 
local averaging of the frequency coefficients. The result 
is extraction of the watermark with very high confidence. 

20 While there has been described and illustrated a 
system for inserting a watermark into and extracting a 
watermark from watermarked data without using an 
unwatermarked version of the data, it will be apparent to 
those skilled in the art that variations and modifications 

25 are possible without deviating from the broad principles 
and teachings of the present invention. 



is considered a noise term. It was not present in the 
prior systems because access to the unwatermarked 
coefficients allowed this term to be removed. The sec- 
ond term a Wj is the original watermark signal which can 
now be detected using conventional correlation. 

Rgure 8 shows an original image before being 
watermarked. Figure 9 is the same image after being 
watermarked in accordance with- the teachings of the 
present invention. 

The watermark in Figure 9 was inserted using a 
gain of a = 0.1 and a chip length of 10.000. The first 
10.000 coefficients of the original image were extracted 
in the sequence shown in Figure 10 to form a one 
dimensional vector. A block average was then com- 
puted over a rectangular window of +/- 3 coefficients. 
The same procedure was applied at the extraction 
stage. 

The correlator responds to randomly generated PN 
sequences with one such sequence being set to the 
originally inserted sequence indicated. A very strong 
and unambiguous response on 0.125 was detected for 
a particular PN sequence. For uncorrected water- 
marks, the correlator output is approximately Normally 
distributed with a variance of l/(N-2). where N is the 
length of the watermark. Thus, for N= 10.000. the stand- 
ard deviation is 0.0 1 and the correlation response of 
0.125 represents over 12 standard deviations. A 
response of approximately 30 deviations was achieved 
with the prior art method using a watermark length of 
only 1.000. The reduction in signal -to -noise ratio is due 
almost entirely to the first term of the right-hand side of 



Claims 

so 1 . A method for inserting a sequence of symbols into 
data to be watermarked comprising the steps of: 

spread spectrum modulating each symbol of 
the sequence of symbols by mapping each 
55 symbol into a PN sequence; and 

embedding each PN sequence in predeter- 
mined coefficients in the data. 

2. The method as set forth in claim 1 . further compris- 
40 ing the steps of: 

obtaining a spectrum of each PN sequence; 
and 

shaping the spectrum to match the spectrum of 
45 the noise. 

3. A method as set forth in claim 2. where said shap- 
ing the spectrum is performed by temporal or spa- 
tial local average of frequency coefficients. 

50 

4. A method of extracting a watermark from water- 
marked data comprising the steps of: receiving 
watermarked data; 

55 spectrum normalizing the watermarked data to 

generate a normalized signal; 
correlating the normalized signal with predeter- 
mined PN sequences corresponding to prede- 
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termined symbols to provide correlated signals 
for each predetermined PN sequence; 
deciding which correlated signal is most likely a 
current symbol; and 

extracting a sequence of most likely current e 
symbols corresponding to the watermark. 
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